Nielsen theory of transversal fixed point sets

Examples exist of smooth maps on the boundary of a smooth manifold M which allow continuous extensions over M without fixed points but no such smooth extensions. Such maps are studied here in more detail. They have a minimal fixed point set when all transversally fixed maps in their homotopy class are considered. Therefore we introduce a Nielsen fixed point theory for transversally fixed maps on smooth manifolds without or with boundary, and use it to calculate the minimum number of fixed points in cases where continuous map extensions behave differently from smooth ones. In the appendix it is shown that a subset of a smooth manifold can be realized as the fixed point set of a smooth map in a given homotopy class if and only if it can be realized as the fixed point set of a continuous one. A special case of this result is used in a proof of the paper.